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A METHOD OF CREATING A SEMANTIC VIDEO SUMMARY 
USING INFORMATION FROM SECONDARY SOURCES 

BACKGROUND OF THE INVENTION 

The present invention relates to a method of summarizing or abstracting 
video and, more particularly, a method for using information related to video 
obtained from a source other than the video to create an audio-video semantic 
summary of video. 

The dramatic increase in the quantity of available video, a trend which is 
expected to continue or accelerate, has increased the need for an automated 
means of summarizing video. A summary of a video could be viewed as a 
preview to, or in lieu of, viewing the complete, unabridged video. Summarization 
could also be used as a basis for filtering large quantities of available video to 
create a video abstraction related to a specific subject of interest. However, to be 
most beneficial the summary or abstraction should be semantically significant, 
capturing major events and meaning from the video. 

There are three broad classes of techniques for creating video summaries. 
A first class of techniques produces a linear summary of a video sequence. A 
linear summary comprises a collection of key frames extracted from the video. 
Groups of similar frames or shots are located in the video sequence, and one or 
more key frames are selected from each shot to represent the content of the shot. 
Shot boundary detection and selection of key frames within a shot are based on 
lower level video analysis techniques, such as frame to frame variation in color 
distribution or temporal positioning of a frame in a shot. While the creation of 
linear summaries can be automated, the extraction of a linear summary is not 
event driven and may only capture a rough abstraction of the video. Linear 
summaries are useful for video sequences where events are not well defined, 
such as home video, but are not well suited to producing meaningful summaries of 
videos containing well defined events, such as videos of sporting events. 



A second summary extraction technique produces a video story board. 
The story board is a graphic presentation of the video comprising a number of 
nodes and edges. Nodes are created by grouping shots usually on the basis of 
some low level visual characteristic, such as a color histogram. The edges 
5 describe relationships between the nodes and are created by human interaction 
with the summarizing system. While story boarding can produce meaningful 
summaries, it relies on human intervention to do so. 

A third summary extraction technique involves the creation of semantic 
video summaries which requires an understanding of the events in the video and, 

1 0 in many cases, some expertise of the domain or subject area portrayed by the 
video. Obtaining this understanding and expertise through automated means 
has, heretofore, been problematic. Smith et al., VIDEO SKIMMING FOR QUICK 
BROWSING BASED ON AUDIO AND IMAGE CHARACTERIZATION, Carnegie- 
Mellon University Tech Report, CMU-CS-95-186, 1995, utilizes detection of 

1 5 keywords in the audio track or closed captioning accompanying a video as a basis 
for locating meaningful video segments. However, it is difficult to select 
appropriate keywords and the selected keywords may be uttered many times as 
part of some general commentary related to the subject of the video without 
necessarily signaling the presence of corresponding meaningful visual images. 

20 What is desired, therefore, is a method of creating meaningful event driven 

video summaries that minimizes the necessity for human intervention in the 
summarizing process. 

SUMMARY OF THE INVENTION 

25 In a first aspect, the present invention overcomes the aforementioned 

drawbacks of the prior art by providing a method of creating a semantic summary 
of a video comprising the steps of identifying a domain of the video; using the 
domain identity to locate information related to the video at a source other than 
the video; extracting data from the infomiation; and extracting content from the 

30 video related to the data. The domain or sphere of activity of the video can be 
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identified from an electronic programming guide or other indexing means. With 
the domain identity known, the summarizing system can link to a worldwide web 
site or an electronic programming guide containing statistics or other summary 
information about events of the video. In this way, the system can obtain the high 
5 level knowledge about the progress of events in the video to permit the system to 
identify content corresponding to significant events. Thus, although the 
summarizing information was created for other purposes it can be used to 
assemble a semantic summary of the video. 

In a second aspect of the invention, a method of abstracting video is 
1 0 provided comprising the steps of locating an index of the video; identifying a 
domain of a video abstraction; using the domain identity to identify in the index 
video for inclusion in the abstraction; and extracting the identified video. An index 
of a video collection is located in an electronic programming guide, on the 
worldwide web or othenwise. When the domain of the particular abstraction is 
1 5 identified, the domain's identity can be used as a basis for searching the collection 
for video relevant to the desired abstraction. 

The method of the present invention provides the high level domain 
knowledge necessary for creating semantic summaries and abstractions by 
locating the knowledge for event recognition and detection in existing sources 
20 without the need for additional human intervention. 

The foregoing and other objectives, features and advantages of the 
Invention will be more readily understood upon consideration of the following 
detailed description of the invention, taken in conjunction with the accompanying 
drawings. 

25 

BRIEF DESCRIPTION OF THE DRAWINGS 

FIG. 1 is a block diagram of the summarization and abstraction system of 

the present invention. 
FIG. 2 is an exemplary worldwide v^eb page containing information related 
30 to a sporting event. 
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FIG. 3 is an exemplary video index containing programming information 
related to headline news. 

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT 

Creation of a semantically significant summary of video footage generally 
requires the ability to recognize and detect events and objects in the visual 
images, together with some high-level knowledge of the domain or sphere of 
activity of the video upon which to base the inference of the occurrence of an 
event in the video. For example, providing a meaningful summary of a sporting 
event requires knowledge concerning the progress of events in the game to permit 
the summarizing system to detect significant plays, such as scoring attempts, and 
the capability to locate and extract a visual representation of the detected event. 
While computer vision techniques are increasingly capable of object recognition 
and detection, event recognition requires a higher level of understanding of the 
progress of events than computers have been capable of achieving by processing 
video. 

The inability of automated techniques to develop a sufficient understanding 
of events in the domain of a video, in such a manner as to enable detection of 
events, is a significant problem in automating the creation of semantically 
significant video summaries. While this remains a problem, the present inventors 
realized that there are automatically accessible sources of infomiation separate 
from the video itself that index or summarize events in the domain of many videos. 
Further, even though this infomiation was created for other purposes, it may be 
used for creating semantic summaries or abstractions of video. The method of 
creating a semantic video summary of the present invention uses available 
information from sources other than the video, such as world-wide-web sites, an 
electronic programming guide, or other sources, to aid in event inference and 
detection to create meaningful summaries. 

Referring to FIG. 1, the summarization method of the present invention first 
identifies a domain 1 or sphere of activity in which a semantic summary system 2 



will operate. Higher level domains include sports, news, movies, etc. These 
domains may be further divided into more specific subject areas, such as 
basketball, soccer, football, etc. The identity of the domain of a broadcast video 
may be extracted from an electronic programming guide that lists television 
5 programming and content for viewers. Other indexing systems related to video 
may also be used to identify the domain. Based on the identification of the 
domain, the summarizing system 2 links to a secondary source of infonnation 4, 
such as a worldwide web site, which includes statistical, summary, or other 
indexing infonnation related to the events which are the subject of the video 6. 

10 Many worldwide web sites include statistics, summary information, or commentary 
related to sports activities, news, or movies. Much of the infonnation at worldwide 
web sites is in the form of textual summaries. Using optical character recognition, 
parsing of hypertext markup language (HTML) code or other known techniques, 
the summarization system 2 extracts information that is semantically significant to 

15 the video 6. 

For example, FIG. 2 illustrates an exemplary textual summary related to a 
soccer game posted on a worldwide web site. From such a textual summary the 
video summarization system 2 can extract information, such as names and times 
of events, and identification of subjects (e.g., names and uniform numbers of 

20 players). Using the data obtained from the summary, the summarization system 2 
can pinpoint a number of shots from the video 6 around the times of the scoring 
events for inclusion in the video summary 8. On the other hand, the system might 
use object detection and optical character recognition to locate the specific 
players to further pinpoint video content corresponding to the events identified in 

25 the textual summary described at the secondary source of information 4. The 
summarization system 2 pennits a user to enter preferences 10 related to the 
summary 8 to be created. For example, a user might instruct the system to 
search out events related to a particular individual or subject. 

The summarization system 2 can also be used to create an abstraction of a 

30 video collection. Utilizing indexing information for the video collection extracted 
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from the secondary source of information 4 and a domain for the video 
abstraction 1 provided by a user, the summarization system 2 can search video 
footage 6 that may be stored in a home file system or filter broadcast video to 
identify portions of the video matching events or subjects in the requested domain. 
5 For example, the summarization system 2 can extract information related to 
source, story title, names of subjects, or times of events from a site summarizing 
televised news programming. 

FIG. 3 illustrates an exemplary textural index of broadcast news events. If 
the user wishes to review video footage related to "DuPont," a keyword search by 
1 0 the summarization system of such a news index at a worldwide web site or in an 
electronic programming guide would locate two relevant news events. From this 
information the system 2 can direct the recording of the appropriate videos for an 
abstraction 8 of the news programming for eventual presentation to the viewer. 
Existing audio or visual information from a secondary source 4, such as a web 
1 5 site, may also be down loaded from the secondary source 4 and matched to the 
video segments obtained from broadcast media. 

All the references cited herein are incorporated by reference. 
The terms and expressions that have been employed in the foregoing 
specification are used as terms of description and not of limitation, and there is no 
20 intention, in the use of such temis and expressions, of excluding equivalents of 
the features shown and described or portions thereof, it being recognized that the 
scope of the invention is defined and limited only by the claims that follow. 
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What is claimed is: 



A method of creating a semantic summary of a video comprising the steps of: 
/ (a) identifying a domain of said video; 
5 (b) using said domain to locate information related to said video at a source 

other than said video; 

(c) extracting a datum related to a semantic event from said information; 
and 

(d) extracting content related to said semantic event from said video related 
10 to said datum. 

2. The method of claim 1 wherein said information is a textual summary of 
events. 

15 3. The method of claim 1 wherein said information is included in a worldwide web 
site. 

4. The method of claim 1 wherein said information is included in an electronic 
programming guide. 

20 

5. The method of claim 1 wherein said domain is identified from an electronic 
programming guide. 

6. The method of claim 1 further comprising the step of selection of said datum 
25 by a user of said summary. 

A . A method of abstracting video comprising the steps of: 
/ (a) locating an index of said video from a source external to said video; 
(b) identifying a domain of said video abstraction; 
30 (c) using said domain to identify in said index video for inclusion in said 
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abstraction; and 
(d) extracting said identified video. 

8. The method of claim 7 wherein said index is included in a worldwide web site. 

5 

9. The method of claim 7 wherein said index is included in an electronic 
programming guide. 

1 0. The method of claim 7 further comprising the step of identification of said 
10 domain by a user of said abstraction. 

14. A method of creating a semantic summary of a video comprising the steps of: 
/ (a) identifying a domain of said video; 

(b) using said domain to locate a textual summary of said video; 
1 5 (c) extracting a datum related to a semantic event relevant to said video 

summary from said textual summary; 

(d) locating content in said video corresponding to said datum; and 

(e) extracting said content related to said semantic event corresponding to 
said datum from said video for inclusion In said summary. 

/ 

i2. A method of abstracting video comprising the steps of: 

(a) locating an index of said video in at least one of a worldwide web site 
and a programming guide; 

(b) identification of a domain of said video abstraction by a user of said 
25 abstraction; 

(c) using said domain to select in said index video for inclusion in said 
abstraction; and 

(d) extracting said Identified video. 

30 
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ABSTRACT OF THE DISCLOSURE 



5 Creating semantic summaries and abstractions of video requires high level 

knowledge of the domain of the video. Obtaining this high level information 
without human intervention is problematic. A method of producing video 
summaries and abstractions is provided where high level knowledge necessary for 
meaningful summarization is obtained from existing sources such as the 
1 0 worldwide web or an electronic programming guide. With this information, the 
summarization system can automatically identify significant events or subjects and 
extract video related to these events or subjects. 
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DECLARATION AND POWER OF ATTORNEY 



As the below named inventors, we hereby declare that: 

Our residence, post office address and citizenship are as 
stated below next to our names. 

We believe that we are' the original inventors of the subject 
matter which is claimed and for which a patent is sought on 
the invention entitled: 

A METHOD OF CREATING SEMANTIC VIDEO SUMMARY USING 
INFORMATION FROM SECONDARY SOURCES 

the specification of which 

[X] is attached hereto. 

was filed on as 

[ ] Application Serial No. 

and was amended on 

(if applicable) 

We hereby state that we have reviewed and understand the 
contents of the above-identified specification, including the 
claim (s) , as amended by any amendment referred to above. 

We acknowledge the duty to disclose information which is 
material to the examination of this application in accordance 
with Title 37, Code of Federal Regulations, § 1.56. 

We hereby claim foreign priority benefits under Title 35, 
United States Code, § 119 of any foreign application (s) for 
patent or inventor's certificate listed below and have also 
identified below any foreign application for patent or 
inventor's certificate having a filing date before that of the 
application on which priority is claimed: 

Prior Foreign Application (s) 

Priority 
Claimed 

[ ] Yes [ ] No 

(Number) (Country) (Day/Month/Year Filed) 

[ ] Yes [ ] No 

(Number) (Country) (Day/Month/Year Filed) 

We hereby claim the benefit under 35 U.S.C. § 119(e) of any 
United States provisional application (s) listed below. 



(Application Serial No.) (Filing Date) 
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(Filing Date) 
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Title 3S, United States CoOm, 5 112. we afiknowledga the duty 
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Of Federal Reg^aations, a 1.56 which occurred between the 
Tii^ Zf^^ prior application and the national or PCT 
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(patented/ pending* 
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i«L°°?^T°*'?''' Second Avenue. Portlind, Oregon 

fZ3 ti^^^F^??^ number 503-237^5631, our attomeya, jointly 
and individually, ho prooeeute thie application and to 
tranaact all touaine«B in the Patent and TrademarJc Office 
connected therewith. 

We hereby declare that all cfcatementa made taexain of our own 
ia^lfoS* BJCB true and that all etatementa made on information 
ana Belief are believed to be true; anti further that these 
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We hereby claim the benefit under Title 35, United States 
Code, § 12 0, of any United States application (s) listed below 
and, insofar as the subject matter of each of the claims of 
this application is not disclosed in the prior United States 
application in the manner provided by the first paragraph of 
Title 35, United States Code, § 112, we acknowledge the duty 
to disclose material information as defined in Title 37, Code 
of Federal Regulations, § 1.56 which occurred between the 
filing date of the prior application and the national or PCT 
international filing date of this application: 
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connected therewith. 
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this application is not disclosed in the prior United States 
application in the manner provided by the first paragraph of 
Title 35, United States Code, § 112, we acknowledge the duty 
to disclose material information as defined in Title 37, Code 
of Federal Regulations, § 1.56 which occurred between the 
filing date of the prior application and the national -or PCT 
international filing date of this application: 



{Application Ser. No.) (Filing Date) (Status) 

(patented, pending, 
abandoned) 



We hereby appoint Jacob E. Vilhauer, Jr., Reg. No. 24,885, 
Charles D. McClung, Reg. No. 26,568, Dennis E. Stenzel, Reg, 
No. 28,763, Donald B. Haslett, Reg. No. 28,855, William O. 
Geny, Reg. No. 27,444, J. Peter Staples, Reg. No. 30,690, 
Kevin L. Russell, Reg. No. 3 8,292, Bruce W. DeKock, Reg. 
No. 40,585, Nancy J, Moriarty, Reg. No. 40,733, Timothy E. 
Siegel, Reg. No. 37,442 and Timothy A. Long, Reg, No. 28,876, 
all of the firm of Chernoff, Vilhauer, McClung & Stenzel, LLP, 
1600 CDS Tower, 601 S.W. Second Avenue, Portland, Oregon 
97204, telephone number 503-227-5631, our attorneys, jointly 
and individually, to prosecute this application and to 
transact all business in the Patent and Trademark Office 
connected therewith. 

We hereby declare that all statements made herein of our own 
knowledge are true and that all statements made on information 
and belief are believed to be true; and further that these 
statements were made with the knowledge that willful false 
statements and the like so made are punishable by fine or 
imprisonment, or both, under Section 1001 of Title 18 of the 
United States Code and that such willful false statements may 
jeopardize the validity of the application or any patent 
issued thereon. 



Dated : 

Full name of sole inventor 

Residence 

Citizenship 

Post Office Address 



Richard Qian 
Camas , WA 
China 

2 844 NW 44^^ Avenue 
Camas, WA 98607-8318 





Residence 

Citizenship 

Post Office Address 



Peter J,L. Van Beek 
Vancouver , Washington 
Netherlands 

12800 NE 4th St, Apt. LL113 
Vancouver, WA 98684 



f 



IN THE UNITED STATES PATENT AND TRADEMARK OFFICE 
CERTIFICATE UNDER 37 CFR ^3.73 (b) 



Applicant : Richard Oian and Peter J.L. Van Beek 

Application No: Filed: Concurrently herewith 

Title : A METHOD OF CREATING SEMANTIC VIDEO SUMMARY USING 

INFORMATION FROM SECONDARY SOURCES 

Sharp Laboratories of America^ Inc. (Assignee) a Washington corporation 

(Type of Assignee, e.g., corporation, partnership, university, government 
agency, etc .) certifies that it is the assignee of the entire right, title and 
interest in the patent application identified above by virtue of either: 



A. [ ] An assignment from the inventor (s) 

identified above. The assignment 

Trademark Office at Reel , 

copy thereof is attached. 



of the patent application 
was recorded in the Patent and 
Frame , or for which a 



B. [ ] A chain of title from the inventor (s), of the patent application 

identified above, to the current assignee as shown below: 

1. From: To: 



The document was recorded in the Patent and Trademark Office at Reel 
Frame , or for which a copy is attached. 



2. From: To: 

The document was recorded in the Patent and Trademark Office at Reel 
Frame , or for which a copy is attached. 



[ ] Additional documents in the chain of title are listed on a supplemental 
sheet . 



[x] Copies of assignments or other documents in the chain of title 

are attached. (Where one or more of the documents is 
unrecorded. ) 

The undersigned has reviewed all the documents in the chain of title of 
the patent application identified above and, to the best of undersigned's 
knowledge and belief, title is in the assignee identified above. 

The undersigned (whose title is supplied below) is empowered to sign 
this Certificate on behalf of the assignee. 

I hereby declare that all statements made herein of my own knowledge are 
true, and that all statements made on information and belief are believed to 
be true; and further, that these statements are made with the knowledge that 
willful false statements, and the like so made, are punishable by fine or 
imprisonment, or both, under Section 1001, Title 18 of the United States Code, 
and that such willful false statements may jeopardize the validity of the 
application or any patent issuing thereon. 



Date : December 6, 1999 



Name : Kevin L. Russell 



Title : Attorney for the Assignee (a copy of the Power of Attorney is 
attached/ 



Signature : 




Chernoff, Vilhaue£, McClung & Stenzel, LLP 

1600 CDS Tower 

602 SW Second Avenue 

Portland, Oregon 97204 

Tel: (503) 227-5631 



